SEPA: Structure Evolution and Parameter Adaptation in Feed-Forward Neural Networks

In developing algorithms that dynamically changes the structure and weights of ANN (Artificial Neural Networks), there must be a proper balance between network complexity and its generalization capability. SEPA addresses these issues using an encoding scheme where network weights and connections are encoded in matrices of real numbers. Network parameters are locally encoded and locally adapted with fitness evaluation consisting mainly of fast feed-forward operations. Experimental results in some well-known classification problems demonstrate SEPA’s high consistency performance in classification, fast convergence, and good optimality of structure. 1 SEPA Strategy Until now, ANN architecture design remains to be one of the most important areas of research due to the lack of general criteria in finding an optimal network topology for a particular class of problems. To address this issue, SEPA uses a GA-based model of ANN where weight adaptation is not gradient-based but stochastic (Fig. 1). To achieve a proper balance in ANN’s network complexity and generalization capability, SEPA’s fitness function considers three important parameters: training error; hidden nodes; and connections. Fig. 1. Encoding scheme E. Cantú-Paz et al. (Eds.): GECCO 2003, LNCS 2724, pp. 1600–1601, 2003. c © Springer-Verlag Berlin Heidelberg 2003 SEPA: Structure Evolution and Parameter Adaptation 1601 Equation (1) is SEPA’s fitness function where: α, β are complexity constants; n,N are no. of active hidden nodes and its total; and c, C are no. of active connections and its total. The degree of influence in network complexity is controlled by setting α and β parameters appropriately. SEPA uses α = 0.70 and β = 0.30. Q = 1 N N ∑ i=1 (Ti −Oi) + α( n N ) + β( c C ) (1) SEPA’s structure evolution is carried out by mutation using (2) and (3) with mutation probability set to 0.01. Structure exploration is carried out by adapting the mutation operator using Gaussian perturbation: m′ = m+N0(0, σ) (2) w′ ij = wij +N0(0,m ′) (3) where σ is the step size parameter (SSP) influencing the strength of mutation. SEPA uses step sizes of 5 (SSP5) and 100 (SSP100) in the experiment. SEPA’s selection criterion uses an elitist roulette-wheel method where only the two best parents are retained. 2 Summary of Results SEPA exhibits very low classification error variability (maximum=0.02) and small network size in all problems (Table 1). The experiments suggest that SSP can serve as a local optimizer in SEPA. For well-behaved classification problems, larger SSP produces smaller networks, faster convergence, and good classification performance. On the other hand, smaller SSP produces larger network structure with slower convergence rate but with good adaptability in noisy data. Table 1. Experiments SSP ave var 100 3.84 1.65 5 5.84 2.71 p=0.00 a)cancer SSP ave var 100 3.04 1.30 5 4.28 3.39 p=0.00 b)iris SSP ave var 100 3.24 1.45 5 4.78 3.60 p=0.00 c)wine SSP ave var 100 5.9 0.10 5 5 0.00 p=0.00 d)glass HIDDEN NODES SSP ave var 100 3.6% 0.00 5 3.8% 0.00 p=0.64 a)cancer SSP ave var 100 7.2% 0.00 5 10.4% 0.00 p=0.01 b)iris